Blog Tags GitHub
All tags

sparse attention

2 posts tagged with "sparse attention"

The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

最大规模的训练无关稀疏注意力经验分析，覆盖Qwen 2.5/Llama 3.1/Gemma 3三大模型家族(4B-72B)、16K-128K序列长度、稀疏度0-0.95，发现稀疏注意力Pareto前沿、prefill/decode两阶段行为差异、序列长度与稀疏容忍度关系

sparse attention training-free isoCost analysis Pareto frontier long-context transformer LLM taxonomy Qwen 2.5 Llama 3.1 Gemma 3 vertical-slash SnapKV Quest Block-Sparse FlexPrefill Ada-SnapKV

ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers

ScalingAttention：通过权重编码的内在稀疏注意力拓扑发现机制（WEST）与保真度感知的稀疏度自适应调节（FAST）以及位级块稀疏注意力内核（crm kernel），实现视频扩散Transformer的高效训练无关推理加速

video diffusion transformers sparse attention intrinsic sparse topology WEST FAST Hellinger distance CRM kernel FlashAttention-3 Wan2.1 HunyuanVideo training-free block-sparse attention